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Abstract 

It is proposed to base the assembly of tests for the measurement of multiple 
abilities on targets for the (asymptotic) variance functions of the estimators of each 
of the abilities. A linear programming model is presented \which can be used to 
computerize the assembly process. Several cases of test assembly dealing \with 
multidimensional abilities are distinguished, and versions of the model applicable to 
each of these cases are discussed. An empirical example of a test assembly 
problem from a t\wo-dimensional mathematics item pool concludes the paper. 
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Assembling Tests for the Measurement of Multiple Abilities 



A standard procedure for assembling tests from an item pool fitting a 
unidimensional IRT model was suggested by Birnbaum (1968). The central 
quantity in his suggestion is the test information function which is defined as 
Fisher’s information about the unknown ability parameter 0 in the responses to 
the test taken as a function over the range of possible values of the parameter. 
For a one-dimensional IRT model of choice, let L(0) be the likelihood statistic 
associated with the responses to the test. The test information function is given by 

l(0) = -«<0<«. (1) 

30 ^ 



Birnbaum’s suggestion was to first design a target for the information function of 
the test and then select items in the test such that the sum of their information 
functions matches the target. The procedure capitalizes on the fact that local 
independence between item responses guarantees additivity of the item 

information functions. If lj(0) is the information function of item i (i=1 n), defined 

analogously to Equation 1 for the likelihood statistic associated with the response 
to this item, it holds that 



n 

1(0) = S lj(0), (2) 

i=1 

where n is the number of items in the test. 

If 0 is the maximum-likelihood estimator (MLE) of 0, 



Var(0|0) ^ 1/1(0) for n ^ « 



( 3 ) 
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(e.g., Kendall & Stuart, 1976, chap. 18). Because of this reciprocity, setting a 
target for the information function is equivalent to setting one for the (asymptotic) 
variance function of the ability estimator. 

However, in practice, in spite of the additivity of the item information 
functions, the problem of picking n items from a pool of a realistic size such that 
the sum of the functions matches the target best over the range of possible values 
of 6 is not a trivial task. The prohibitively large number of possible combinations 
rules out optimal test assembly by hand. In fact, even for a high-speed computer 
explicit enumeration of all possible solutions and picking out the best is an 
unrealistic job. The problem gets more difficult still if the test has to meet various 
constraints on the selection of the items related to the distributions of, for example, 
item content, item format, or the values of certain item parameters. To implement 
Birnbaum’s procedure, efficient algorithms which reduce the set of feasible 
solutions to a smaller set of candidate solutions and then select an optimal one are 
badly needed. 

ApPt'jcation of Linear Programmino 

Formally, the problem of test assembly is an instance of the problem of 
constrained combinatorial optimization which, in its mathematical generality, has 
been studied in such fields as applied mathematics, decision theory, and 
operations research. It should therefore not come as a surprise that attempts to 
implement Birnbaum’s procedure in a computer algorithm have been based on 
techniques of combinatorial optimization, in particular on techniques of (mixed) 
integer programming from the field of Linear Programming (LP). Though 
suggestions to resort to LP for solving test assembly problems were made earlier 
(Feuerman & Weiss, 1973; Votaw, 1952; Yen, 1983), the first LP model for a 
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variation of Birnbaum’s procedure was published in Theunissen (1985). Ever since, 
modeling various test assembly problems as an LP problem and finding algorithms 
and heuristics to solve the model for an optimal solution has been a fruitful field of 
research. Some references to relevant papers are: Adema (1990, 1992); Adema, 
Boekkooi'Timminga, and van der Linden (1991); Adema and van der Linden 
(1989): Amstrong and Jones (1992); Amstrong, Jones, and Wu (1992); Boekkooi- 
Timminga (1987, 1990); Timminga and Adema (1995); van der Linden (1994); van 
der Linden and Boekkooi-Timminga (1988, 1989); van der Linden and Luecht (in 
press). Important heuristic approaches to the same problems have been presented 
in Ackerman (1989), Luecht and Hirsch (1992), and Swanson and Stocking (1993). 

Maximin Model 

The model taken as a starting point for the problem of multidimensional 
test assembly is the maximin model for unidimensional assembly in van der Linden 
and Boekkooi-Timminga (1989). It is assumed that a test of n items has to 
measure an interval of possible 0 values with uniform accuracy, and that the test 

assembler wants to control this behavior at the ability points 0|(, k=1 K. To 

attack the problem, decision variables Xj, i=1 ,...,! are defined, one for each item in 
the pool, which take the value 1 if the item is included into the test and the value 0 
otherwise. The maximin model is as follows: 

maximize y (4) 

subject to 




I 

Z lj(0k)xj - y > 0, 
i=1 



k=1,...K, 



(5) 
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Z Xj = n , 
i=1 




(6) 


Xj G (0,1), 


i=1 1. 


(7) 


y > 0 . 




(8) 



The model is based on the idea that a common lower bound y to each of the 
values of the test information function defined by the inequality in Equation 5 
should be maximized, as is done by the objective function in Equation 4. At the 
same time, Equation 6 constrains the length of the test to size n. The last two 
equations define the ranges of values of the decision variables in the model 

The model can be generalized to a target for the test information function 
of any shape by providing the variable y in Equation 5 with coefficients r|^ which 
govern the relative heights of the test information function at 0 i-6k (van der 
Linden & Boekkooi-Timminga, 1989). For ease of exposition only the case of a 
uniform target will be considered in this paper. Also, a catalogue of adoftional 
linear constraints is available to model test specifications with respect to sich 
categories as item content, format, response time available, the values of classical 
or IRT item parameters, interdependencies between test items, etc. (van de Linden 
& Boekkooi-Timminga, 1989). For an illustration of the use of some of the 
constraints, see the empirical example below. The model has been implemented 
as one of the options in the computer program CONTEST (Timminga & van der 
Linden, 1995) which contains a large choice of algorithms and heuristics to solve 
the model for an optimal combination of values for its decision variables. Quick 
heuristics to solve certain test assembly problems have been presented in 
Ackerman (1989) and Leucht and Hirsch (1992). If the model has a network flow 
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structure, computation of an optimal solution simplifies dramatically (e.g., 
Amstrong, Jones & Wu, 1992). 

It is the purpose of this paper to present models for the optimal assembly 
of tests measuring more than one ability. However, unlike the case of an IRT 
model with a single ability parameter, for a model with multiple ability parameters 
Fisher’s information measure is no longer a scalar but a (non-diagonal) matrix. 
Also, the (asymptotic) variances of the MLEs of the ability parameters are not 
given by the reciprocals of the diagonal elements of the information matrix, but by 
the diagonal elements of the variance-covariance matrix, which is the inverse of it. 
Hence, the motivation to use a target directly for Fisher’s information measure fails 
for the case addressed in this paper. To solve the problem, the use of targets for 
the variance functions in the model will be explored. Then a generalization of the 
maximin model and a heuristic for the assembly of tests in the presence of multiple 
ability parameters will be proposed. Next, various cases of multidimensional test 
assembly will be discussed, and it will be shown how the model applies to each of 
these cases. The cases include such examples as the assembly of a test 
measuring two intentional abilities, a test insensitive to a nuisance ability, or a test 
required to have a simple structure of underlying abilities. The paper concludes 
with an empirical example of the use of the model to a test assembly problem. 



Multidimensional Test Assembly 

The multidimensional IRT model considered in the paper is the logistic model 
discussed by McKinley and Reckase (1983), Reckase (1985; in press), and 
Samejima (1974). To simplify notation, only the case of two ability parameters, 
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(01,02), is considered. Let the response variables Ujj take the value 1 if the 

response of person j=1 N to item i=1 n is correct and the value 0 otherwise. 

The model is defined by the following logistic response function: 

Pi(01.02) = P(Ujj|aij,a2j,dj,0i,02) 

exp(aij0i + a2j02 + dj) (^) 

1 +exp(aij0i +a2j02+dj) 

where (aij,a 2 j) are the discrimination parameters of item i along the abilities 0i 
and 02 , respectively, and dj can be interpreted as a composite parameter 
representing the difficulty of the item. In the remainder of this paper, it is assumed 
that these item parameters are known and that the model is used to estimate the 
abilities ( 0 ij, 02 j) fi’cnn a realization of the response variables U|j=Ujj for i=1,...,n 
and j=1 N. 



Variance Functions 

For the case of two ability parameters Fisher’s information matrix is 
defined as: r n 



1 ( 01 , 02 ) 



a^lnL 


a^lnL 


30^ 


D0-|D02 


InL 


InL 


dQ]dQ2 


a02 



( 10 ) 



where L now is the likelihood statistic associated with the data under the model in 
Equation 9. Following the derivation in Ackerman (1994, Appendix) and using the 
notation PjsP|( 0 i, 02 ), the following result is obtained for the model in Equation 9: 
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1 ( 01 , 02 ) = 



n 



n 



X a^jPjQj X aija2jPjQj 

i=1 i=1 

n n 2 

Xaiia2iPiQj X a2jPjQj 

i=1 i=1 



( 11 ) 



Standard techniques for matrix inversion yield the variance-covariance 
matrix of the MLE’s of (0i,02): 

no n 



X a2jPjQj 
i=1 



-X aiia2iPiQj 
i=1 



V(0i.02|0i,02) = 




where 



11 ( 01 . 02)1 



11 ( 01 . 02)1 



( 12 ) 



!I(01.02)| = ( X afjPjQj)( X a^jPiQj) - ( X ai;a2jPjQi)2 
i=1 i=1 i=1 



( 13 ) 



is the determinant of the matrix in Equation 11, which is assumed to be n e nz e r o 
throughout this paper. The diagonal elements of the matrix in Equation 12 are the 
(asymptotic) variances of the MLEs of 0i and 02, respectively: 



Var(§i |0i,02) = ( X a 2 jPjQj)[( X afjPjQj)( X a 2 jPjQj) 
i=1 i=1 i=1 

^ 2 1 

-(2:aija2iPiQi)T\ 

i=1 

Var(02|0i,02) = ( X afjPjQj)[( X afjPjQi)( X aljPjQj) 
i=1 i=1 i=1 

■ " 2 1 

-(2:aiia2iPiQi)T • 
i=1 
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( 15 ) 
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Observe that Equation 14 gives the (asymptotic) variance of 0i for the true ability 

of the examinee being a point in a t\wo-dimensional space. Thus, the variance of 

01 not only depends on the true value of 0i but also on the value of 02- The 

same holds for the (asymptotic) variance of 0? in Equation 15. Also, observe that 

2 2 

the t\wo variances differ only by the factors a 2 \ a.|j in the t\wo numerators. 

If the variances in Equations 14-15 are taken as functions over the 
complete t\wo-dimensional ability space, t\wo variance functions are defined, one f or §i 
and the other for 02- Figure 1 sho\ws the plots of three pairs of variance functions 
each for a different 



[Figure 1 about here] 

test. The first test consists of nine items \which have larger values for the first than 
for the second discrimination parameter: a^=(2.0, 2.0, 2.6, 1.2, 1.5, 1.7, 1.2, 0.8, 
0.9): 32=(0.1, 1.1, 1.7, 2.4, 2.0, 3.0, 1.9, 2.1, 1.8); and dj=0.0 for all items. In the 
second test, the values for the t\wo discrimination parameters are equal on average 
for the six items in the test: a.|=(1.8, 2.6, 1.7, 1.8, 2.2, 2.0), a2=(2.0, 1.8, 1.9, 1.7, 
1.8, 1.7): and dj=-2.0 for al items. The six items in the third test have values for the 
first discrimination parameter exactly twice the ones for the second parameter, the 
only exception being Item 6 for which the values slightly deviate from this 
proportion: a^=(2.0, 2.0, 2.6, 2.4, 2.0, 3.0): a2=(1.0, 1.0, 1.3, 1.2, 1.0, 1.7): and 
dj=0.0 for all items. The result of this case of 'weak identifiability' is a variance 
function for 0i which is low only locally along a line in the ability plane and a 
function for §2 which never takes on any small value. (Note that for readability in 
all three figures the surfaces are cut at a height of 100.) 
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Targets for Variance Functions 

It is proposed to define targets for the two variance functions to guide the 
multidimensional test assembly process. Graphically, the proposals means that 
tests are assembled such that the plots of their variance functions meet previously 
defined forms. For example, if ability 0-| is considered to be more important than 
02, a target for Var(0-| |0-|,02) uniformly lower than the one for Var(02|0-|,02) 
over the ability area of interest makes sense. The proposal implies that the 
covariance function in Equation 12 can be ignored. This implication is in 
agreement with the fact that test assembler are typically not interested in 
covariances between ability estimators. 

Computational Complications 

Test assembly with simultaneous targets for two distinct functions is an 
example of a multi-objective decision problem. Standard approaches to decision 
problems with two objectives are, for example, to combine the two objectives into 
one objective function or to focus on one as the objective function and represent 
the other by a constraint with an optimally chosen bound. More important, 
however, is the fact that the two expressions in Equations 14-15 are nonlinear. A 
realistic objective function based on the difference between the two expressions 
and their targets will also be nonlinear. Due to this complication, algorithms 
allowing for optimal multidimensional test assembly which operate in polynomial 
time are not available. Hence, unless the problem is trivially small, the use of a 
heuristic which yield good but not necessarily best solutions seems the only 
possibility left. 
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Multidimensional Maximin Model 

Further analysis of the variance functions in Equations 14-15 reveals that, though 
nonlinear, they consists of sums each of which is additive in the items. The role of 
these sums becomes more obvious if decision variables are added to Equation 1 4, 
and this variance function of 0i is written as: 

Var(0i |0i,02) = [ I a^jPjQjXj - ( Z aija2iPjQiXj)‘^/( Z aJPjQiXi)]'^ . (16) 

i=1 i=1 i=1 

It is now immediately clear that, for a fixed value of (0i,02), the function in 
Equation 16 decreases in value if the values of the decisions variables Xj, i=1 



are chosen such that 






* 2 

ZaijPjQjXi 

i=1 

I 


increases; 


(17) 


Z ogjPPjXj 
i=1 
1 


increases; 


(18) 


1 

Z a-]ja2jPjQjXj 


decreases. 


(19) 



,i=1 

However, note that for a fixed set of item parameter values, the 
expression in Equation 19 can not decrease independently of the expressions 
Equations 17-18. In fact, a tradeoff exists between these two sets of expressions 
because any choice of parameter values which decreases the last expression also 
decreases the first two. The optimum value of Equation 16 thus depends on the 
relative rates of change of the three expressions. This fact suggests an approach 
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in which the expression in Equation 19 is minimized for a systematically varying 
series of lower bounds on the expressions in Equations 17-18. 

Consider the following variant of the maximin model in Equations 4-7 in 

which, for a selection of ability points (0ip.02q). P=1 P. q=1.-.Q. minimization 

of the expression in Equation 19 is taken as the objective function and the 
expressions in Equations 18-19 are constrained by lower bounds: 



minimize y 



subject to 

I 

Z 3i j32iPi(®1p>®2q)^i(®1p’®2q)^i 
i=1 

’ 2 

Z 3.| jPj(0^p,02q)Qj(0ip,02q)Xj ^ c^, 

i=1 
’ 2 

Z 32jPj(0ip,02q)Qj(0ip.02q)Xj ^ C2, 
I 

Z Xj = n. 
i=1 

X e {0,1}, 



y>0. 



( 20 ) 



P=1 P. q=1 Q. (21) 

P=1 P. q=1 Q. ( 22 ) 

P=1 P. q=1 Q.(23) 

(24) 

i=1 1. (25) 

(26) 



The basic idea is now to run this model systematically varying the values of c.| and 
C 2 until optimal variance functions are found. 
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Choices of Values for and C o 

First, note that the following inequalities hold: 




i=1 max 

' 2 ^^ ^ ^ 2 



(27) 



0 < Z a2jPjQjXj < .25 Z a2j , 
i=1 max 



(28) 



where the right-hand sums are taken over the n items with the largest values for 
a^j and a 2 j in the item pool, respectively. Thus, the right-hand sides can be used 
as upper bounds for c^ and C 2 - Observe, however, that items with high values for 
a,j are not necessarily the ones with high values for a 2 j and vice versa; therefore 
these bounds will seldom be reached in practice. 

Second, if c.| and/or C 2 are set high, overconstraining may occur and no 
feasible solution found. If infeasibility is found for certain values of c.| and C 2 , no 
larger values have to be tried since these will also yield infeasibility. 

Third, for brevity, let 




i=1 



(29) 




i=1 



(30) 



w = Z aija2jPjOj . 
i=1 



(31) 



Suppose no dependences existed between u,v, and w. The following partial 
derivatives then show the impact of u and v on the variance function of O'] : 
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and 



avar(0i|0i,02) 

3u 



avar(0i|0i,02) 

0v 



-V 



p p 

(uv-w ) 



-w 

(uv-w ) 



(32) 



(33) 



As u.v.w > 0, the derivatives are negative for all possible values of u, v, and w 
(provided uv^^w). Consequently, as already assumed in Equations 17-18, for a 
fixed value of (0'|,02), Var(0-| |0'|,02) is minimal for u and v maximal. However, 
in the model, w is minimized, and the derivatives in Equations 32-33 show that if 
an optimal solution is approached, the marginal contribution of v toVar{0^ !6i,02) 
is likely to be smaller than the one of u. If w approaches zero, the contribution of v 
even becomes negligible. By symmetry, the reverse conclusion holds for the 
contributions of u and v to the variance function of § 2 . These relations seem to 
suggest that a larger value of c^ relative to C 2 favors minimization of 
Var{0^ |0^, 02 ) whereas a smaller value of c^ favors minimizaiion of 
Va‘(02;0i,02). However, the actual problem is one of combinatorial cc" 



It** * I 



over a finite pool of possible values for the item parameters. Also, as already 
noted these values create dependencies between the expressions in Equations 17 
through 19. It is therefore recommended to always check this suggestion for the 
actual item pool in use. For an empirical example, see the analyses further on in 
this paper. 



A Heuristic 

The following heuristic can be used to find a (nearly) optimal solution to 
the test assembly problem: 
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1. Choose a grid of values for {0-|p,02q) which covers the ability 
area of interest. As the variance functions are well-behaved 
smooth functions, a 3x3 or 4x4 grid will generally do. There is no 
need to space the points evenly or to have the same numbers of 
point along both dimensions. 

2. Choose a series of values for (c.|,C 2 ) covering the range of 
possible values below the upper bounds in Equations 27-28, 
taking into account the distribution of the values of the item 
parameters in the pool as well as the goal of the test (see the 
following section) ; 

3. Solve the model in Equations 20-26 using, for example, one of 
the algorithms or heuristics in CONTEST (Timminga & van der 
Linden, 1995); 

4. Calculate the two variance functions for each solution in the 
previous step; 

5. Based on an inspection of the results, repeat steps 3-4 for a finer 
grid of values for (c.|,C 2 ) in the neighborhood of the value for 
which the best variance functions were obtained; 

6. Repeat steps 5 until favorable variance functions are obtained. 
Based on experiences with a previous test assembly problem for a given item 
pool, the first selection of values for (c.|,C 2 ) can be made more effective. For 
example, if infeasibility was met for certain values of (c.|,C 2 ) in a previous run, it 
makes no sense to use larger values for (c.|,C 2 ) in any later test assembly problem 
for the same item pool whatever other constraints are added to the model. An 
implementation of the heuristic for the case of two flat variance functions will be 
given in the empirical example below. 
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Different Cases of Multidimensional Test Assembly 

Five different cases of test assembly are considered in which multidimensionality 
of the item pool plays a role. For each case a different use of the multidimensional 
model in Equations 20-26 is proposed, with the exception of one case which leads 
to the use of a modified version of the unidimensional model in Equations 4-8. The 
main criteria used to classify the five cases are: (1) whether the abilities are 
intentional or should be viewed as 'nuisance abilities': and (2) whether or not the 
abilities underlying the test should display a 'simple structure'. 

1. Two intentional abilities . In this case, the test items are designed to 
measure two abilities, and it is intended to report scores on both abilities for each 
examinee. Thus, for each possible ability point (0i,02) the test should produce 
variances of 0^ and §2 which meet realistic targets. 

The model to be used in this case is the multidimensional maximin model 
in Equations 20-26 possibly provided with additional (linear) constraints to deal 
with remaining test specifications. As already suggested, the relative sizes of the 
Values of c^ and C 2 can be used to control for the importance of the two variance 
functions. 

2. One intentional and one nuisance ability . The test items in the pool are 
designed to measure one intentional ability but happen to be sensitive to another 
ability. When scoring the test, the nuisance ability is ignored and only a score for 
the intentional ability is reported. An obvious example is the case of a nuisance 
ability which creates 'differential item functioning' because a focal and a reference 
grpup have different distributions on it. Removing the effect of the nuisance ability 
by fitting a two-dimensional IRT model and scoring only for the intentional ability is 
believed to be a strategy yielding more informative ability estimates than simply 
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removing all items sensitive to the nuisance ability from the test. 

The best approach in this case seems to ignore the variance function for 
the estimator of the nuisance ability and set a target for the intentional ability only. 
If 02 is the nuisance ability, this approach is implemented if the same model as in 
the previous case is applied, but now with C 2 small relative to c.| . Again, additional 
linear constraints can be added to the model to deal with other test specifications. 

3. One composite ability . This case arises if both abilities are intentional 
but an estimates of the linear combination +P202- Pi- P2^^- 

be reported. A practical motivation for this case might be that the construct 
measured by the test is truly two dimensional but that test consumers want a 
single score equally reflecting both abilities. The variance function of the estimator 
of the linear composite is equal to 

var(pi0i +p2e2|0i,02) = p^Var(9i |0i,02) + p2Var(02|0i,02) 

( 34 ) 

+ 2pip2Covar(0^, 0210-1 ,02). 

(Ackerman, 1994, Eqs. 15-16). Though this function is also an expression 
consisting of the sums of the elements in the information matrix in vj, 

analysis of the expression shows that it misses the monotonicity which could lead 
to a conclusion as in Equations 17-19. The best solution in this case, therefore, is 
to rotate the ability space such that in the reparameterized model the composite 
corresponds to the first ability dimension. Henceforth, the case is identical to the 
previous one. 

4. Simple ability structure: Case I . The item pool is again assumed to 
measure two intentional abilities but the test has to be assembled such that one 
subtest is maximally informative on ©2 and another subtest on 02. This case may 
arise if for diagnostic purposes test performances have to be reported at item level 




21 



Assembling Tests 
20 



and it is thus necessary to know which items measure 0i best and which items 

02 . 

Let n^ be the number of items required to be informative on 0.| and n 2 
the number of items informative on 02- An obvious approach in this case seems 
to apply the multidimensional model in Equations 20-26 first to assembly n.| items 
under the condition of c.|>C 2 and a second time to assemble r \2 items under the 
reverse condition of c.|<C 2 removing the items already selected from the pool. 
However, a clear disadvantage of a sequential approach is that some of the items 
fitting the constraints of the second subtest better have already gone into the 
former. Also, it is not possible to directly constrain item selection with respect to 
item content, format, etc., at the level of the complete test. 

A more favorable solution, therefore, is to select the two subtest 
simultaneously. This choice leads to an adaptation of the multidimensional model 
in Equations 20-26. New decision variables Xjg are introduced which take the value 
of 1 if item i is assigned to subtest s and the value 0 otherwise (s=1,2). The 
adapted model is: 

minimize y (35) 

subject to 

2 I 

Z Z 3ii32iPi(®ip'02q)^i(®1p'®2q^^is “V — Q=1 Q. (36) 

s=1 i=1 

2 I 2 

^ ^ ®1i^i(®1p’®2q)^i(®1p’®2q)^is “ P. q-1....iQ. (37) 

s=1 i=1 
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2 I 2 

Z Z 32i^i^®1p’®2q)^i(®1p'®2q)^is - ^2> 
s=1 i=1 

1 

^ ^is “ ^S’ 
i=1 

2 

Z Xjs ^ 1, 

S=1 

X G {0,1}, 



P=1 P. q=1 Q. (38) 



s=1,2, (39) 



i=1 1. (40) 

i=1 1; (41) 



y>0. 



(42) 



New constraints in the model are the ones in Equation 39 which define the lengths 
of the two subtests as well as those in Equation 40 which prevent the items from 
being assigned to both subtests. The model can be solved using the algorithms 
and heuristics referred to earlier. However, the doubling of the number of decision 
variables generally has an effect on the speed of the algorithms and heuristics 
comparable to the one of doubling the size of the item pool, and, as a 
consequence, some of the heuristics slow down considerably. 

5. Simple ability structure: Case II . For completeness’ sake, the case of 
two subpools of items each fitting a unidimensional IRT but with the complete pool 
fitting only a two-dimensional model is mentioned. The practical motivation for 
assembling a test with this simple structure for its ability space is the same as the 
one in the previous case. 

Again, a simple solution would seem to assemble the two subtest 
sequentially but the same objections to sequential assembly as before apply. A 
model for simultaneous assembly can be obtained by adapting the model in 
Equations 4-8 analogously to Equations 35-42. 
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Empirical Example 

Data from an ACT Assessment Program Mathematics Item Pool were used to 
assemble a test. The pool consisted of 176 items to which a two-dimensional 
version of the model in Equation 9 showed an acceptable fit. The items in the pool 
were classified according to content (PG: Plane Geometry: PA: Pre-Algebra; EA: 
Elementary Algebra; CG: Coordinate Geometry: TG: Trigonometry: lA; 
Intermediate Algebra) as well as skill (BS: Basic Skill; AP: Application; AN: 
Analysis). It was attempted to assembly two tests with flat variance functions for 
both ability estimators over the complete grid of points defined by , 02 = -2, -1, 
0, 1. 2, where measurement of both ability variables was assumed to be inte.niional 
and equally important. One test was assembled using the basic model in 
Equations 20-26 (Model I). The other test was assembled adding the following set 
of constraints to the previous model to simulate the presence of content and skill 
specifications in the assembly program (Model II): 



S Xj>5. 
ieVpQ 


(43) 


Z Xj>5, 
ieVpA 


(44) 


Z Xj > 5, 
i^VEA 


(45) 


Z Xj>5, 
ieVcG 


(46) 


2 Xj > 5, 
ieV-pG 


(47) 
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2 Xj > 5, (48) 

i^V,A 

5: Xj > 15, (49) 

^ X| > 15, (50) 

ieVAp 

5^ X| > 5. (51) 



where, for example, Vpg is the set indices of the items with content classification 
Plane Geometry. For both models test length was set at n=50. The two models 
were solved using the First Acceptable Integer Solution algorithm as implemented 
in the CONTEST program (Timminga & van der Linden, 1995, sect. 6.6; see also 
Adema, Boekkooi-Timminga & van der Linden, 1991). This algorithm was used to 
find the first integer solution with a value for the objective function within 5% from 
the optimal value for the fully relaxed model. Since the two variance functions were 
assumed to be equally important, the values of c^ and C2 in Equations 22-23 were 
set equal to each other. 

For both models values of c.|=C2 larger than or equal to 1.4 led to 
overconstraining and no feasible solutions was found. Therefore, the two models 
were run for c^=C2=0.0(0.1)1.3. The results are summarized in Table 1. As the 
variance functions had to be both low and flat, the mean value (n) plus one 
standard deviation (a) of the 



[Table 1 about here] 

values of the two variance functions over de 25 points of the grid of (6^,82) 
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values \was used as a summary measure. For both models the solutions for the 
smallest values of 0^=02 not only yielded the same value for this measure but also 
contained the same set of items. The best solution for Model I was obtained for 
c^=2=1.0 Plots of the variance functions Var(0^ |0^,02) and Var(02|0i,02) 
associated with the items in this solution are given in Figure 2. Both functions 
show a flat surface over the ability space considered, albeit the one for 0^ 

[Figure 2 about here] 

has a tendency to slightly increase for 0^ approaching the value of 2.0, whereas 
the one for §2 goes up for 02 approaching -2.0. For Model II, the best solution 
was obtained for =02=0.9. Both the numerical results in Table 1 and the plots of 
the two variance functions for the solution obtained for this value show that adding 
the additional constraints in Equations 43 through 51 to the model hardly 
deteriorates the results. 

To assess the numerical effects of setting lower or higher than the 
Value of C 2 , solutions for Model I were computed over the full range of possible 
values for C 2 both for c^=0.2 and c^=1.2. Observe that these two value for are 
near the extremes of the range of values in Table 1 for which feasible solutions 
were obtained. The results are presented in Table 2. The general conclusion from 
this table is that the lower value for c^ favors minimization of the variance function 

[Table 2 about here] 

for §2 both in terms of its average value and spread, whereas the higher value of 
c^ favors minimization of the function for 0^ . These results are as predicted. 
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However, surprisingly, none solutions was better than the one in Table 1 for 



The software in the examples was run on a PC with a 486 processor (66 
Mhz). None of the runs took more than 2 seconds of computing time to reach a 
solution. 



The choice to base the assembly of tests measuring multiple abilities on the 
variance functions associated with the ability estimators seems obvious. However, 
as indicated earlier, the choice involves a multi-objective decision problem with 
nonlinear objective functions. The current paper offers a model along with a 
heuristic scheme to solve the problem. Implementations of the heuristic for other 
targets than the one for the case of two intentional abilities in the emoirical 
example above still have to be examined. It is not unlikely that practical experience 
with the heuristic will reveal that, for some of the cases discussed abCT.-e, certairi 
patterns of item parameter values guarantee optimal variance functions. If so, this 
knowledge, in turn, could be used to further improve the focus of the heuristic. 




Discussion 
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Table 1 

Values of u and a for selected values of c ^=C o for Model I and Model II 



Cl -02 




Model 1 
o 


ji+a 


Model II 
\l o 


[l+C 


0.0 


1.551 


0.833 


2.384 


1.586 


0.922 


2.508 


0.1 


1.551 


0.833 


2.384 


1.586 


0.922 


2.508 


0.2 


1.551 


0.833 


2.384 


1.586 


0.922 


2.508 


0.3 


1.551 


0.833 


2.384 


1.586 


0.922 


2.508 


0.4 


1.551 


0.833 


2.384 


1.562 


0.833 


2.395 


0.5 


1.386 


0.505 


1.891 


1.387 


0.508 


1.895 


0.6 


1.311 


0.373 


1.684 


1.335 


0.384 


1.719 


0.7 


1.286 


0.369 


1.655 


1.277 


0.363 


1.640 


0.8 


1.180 


0.313 


1.493 


1.189 


0.322 


1.511 


0.9 


1.057 


0.295 


1.352 


1.085 


0.294 


1.379 


1.0 


1.037 


0.281 


1.318 


1.104 


0.294 


1.398 


1.1 


1.169 


0.320 


1.489 


1.231 


0.325 


1.556 


1.2 


1.500 


0.437 


1.937 


1.479 


0.410 


1.889 


1.3 


1.826 


0.486 


2.313 


1.907 


0.509 


2.416 


1.4 




Inf 






Inf 





Note . "Inf” means no feasible solution available. 
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Table 2 

Values of |ii and ^2 c^=0.2 and c^=1.2 (Model I) 



C2 


^^1 


Or 


0.2 


G2 


^^1 


Ci= 

<^1 


1.2 


02 


0.0 


2.029 


0.893 


1.073 


0.038 


1.296 


0.265 


3.472 


0.516 


0.1 


2.029 


0.893 


1.073 


0.038 


1.296 


0.265 


3.472 


0.516 


0.2 


2.029 


0.893 


1.073 


0.038 


1.296 


0.265 


3.472 


0.516 


0.3 


2.029 


0.893 


1.073 


0.038 


1.296 


0.265 


3.472 


0.516 


0.4 


2.029 


0.893 


1.073 


0.038 


1.296 


0.265 


3.472 


0.516 


0.5 


2.029 


0.893 


1.073 


0.038 


1.296 


0.265 


3.472 


0.516 


0.6 


2.029 


0.893 


1.073 


0.038 


1.235 


0.230 


3.085 


0.373 


0.7 


2.029 


0.893 


1.073 


0.038 


0.974 


0.102 


2.069 


0.123 


0.8 


2.029 


0.893 


1.073 


0.038 


1.010 


0.110 


1.698 


0.081 


0.9 


2.412 


1.624 


1.014 


0.042 


0.923 


0.096 


1.456 


0.063 


1.0 


2.663 


2.330 


0.949 


0.042 


0.981 


0.107 


1.441 


0.066 


1.1 


7.123 


2.449 


1.426 


0.062 


1.212 


0.166 


1.441 


0.091 


1.2 


12.819 


1.650 


2.036 


0.123 


1.428 


0.232 


1.572 


0.141 




11.606 


2.461 


1.951 


0.140 


1.564 


0.276 


1.533 


0.146 


\a 


8.520 


1.769 


1.661 


0.088 


2.230 


0.373 


1.889 


0.243 


1.5 


7.193 


1.254 


1.643 


0.107 




Inf 




i!e 


4.554 


1.047 


1.470 


0.121 




Inf 




1.7 




Inf 






Inf 





Note . "Inf" means no feasible solution present. 
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Figure 1 . 
Figure 2. 



Figure Captions 

Three examples of variance functions for 0-| and 02 (for the 
values of the item parameters, see the text). 

Variance functions for the tests assembled under Model I (upper 
panel) and Model II (lower panel). 
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used in the empirical example. 
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